DOCOHBIT BBSOIB 



BD 179 250 



IB OOB 177 



AUIHOB 
TITLE 

IKSTITUTION 

5P0MS AGENCV 

PUB DATE 
CONTRACT 
NOTE 

AVAILABLE FfiOM 



EDHS PBICE 
OESCBIPTOfiS 



Danlelf Bvelyn H. : Ely, Donald P. 
Assessing the Conpetencies.of Media Pzofessionals : 
Model for DeterniniDg^ Costs asd Effectiveness. 
Syracuse Oniv«, N.Y* EBIC ClearinQhcuse on 
Informatien Besources. 

National Inst, of Education (DHEH) , Washington, 
D.C. 

79 

UOO-77-0015 

6Bp, ; For a related dccuaent, see £0 ^^9 mO 
Syracuse Oniversity Printing Services, 125 College 
Place, Syracuse, NY 13210 (1^-39; S4.75 prepaid) 

MF0iypC03 Plus Postage. 

>*<Coapetence: ^^Cost Effectiveness; Higher Education; 
>*<Media Specialists: ^^iper forma nee Based Education; 
♦student Evaluation 
Information Analysis Products 



ILiNTIPIERS 
ABSTRACT 

This is a sun nary of the procedures and outcomes of a 
study undertaken to develop alternative strategies for evaluating the 
competencies of library media personnel in competency-based programs. 
The study vas prompted by a realization that, although 
competency-based programs for school library media specialists are 
increasingly being adopted by state agencies as a basis for 
certification and by educational Institutions as a procedure for 
educating media professionals, the effort in deyeloping these 
program? has been heavily weighted touard the generation of 
competency statements with little attention being paid to the perhaps 
more difficult problem of competency assessment. The oethodology for 
this project involved working back and forth between literature 
review and theoretical model development, and the collection of 
empirical data testing various aspects of the model with formal and 
informal interviews of various experts to critigue the developing 
model. A cost-effectiveness approach was selected as the most 
appropriate method, with the ultimate goal of. providing educational 
decision-makers with an array of assessment strategies for each 
individual competency, together with the costs and effectiveness 
measures associated with each strategy. (Author/RAO) 
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COMPETENCY ASSESSMENT STRATEGIES 



This is a summary of the procedures and outcomes of a study under- 
taken by Drs. Daniel and Ely, entitled "Developing Alternative Strate- 
gies for Evaluating Competencies of Library/Media Personnel in Compe- 
tency-Based ProgrciiTis." 

The study was prompted by a realization that although competency- 
based programs for school library media specialists are increasingly 
being adopted by state agencies as a basis for certification and by 
educational institutions as a procedure for educating media profession- 
als, the effort in developing these programs has been heavily (indeed, 
almost exclusively) weighted towards the generation of competency 
statements with little attention being paid to the perhaps more diffi- 
cult problem of competency assessment. The American Association for 
Sthool Librarians (AASL) Certification Committee observes that 



...the art of competency assessment is woefully behind... 
Our lack of experience in the task of perfoVmance assess- 
ment, the identification of data gathering techniques — 
Including what data will be required and under what con- 
ditions, the validation of assessment criteria, the need 
for competent evaluators to implement the assessment 
process, the cost of candidate assessment in terms of 
time and dollars, and reaching consensus on the levels of 
rert if Iration all pose serious concerns for the profes- 
sion. (Cnnaidate Assessment Process for Professional 
Mel in Personnel, 1977) 



Ti) sdmo extent, the general lack of attention to evaluation is an 
artifact of the sequential process of development. Competency-based 
educMtion for scliool media specialists is still a new area. Competen- 
cies must fir5;r be identified and agreed upon before tliere can be any 
coneern with whetfjer or not a person has achieved a given competency 
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or jset of competencies. The AASL Assessment Process Model makes the se- 
quential process clear by beginning in matter-of-fact fashion with 
•'Competence Specification/' (See Figure 1 - Assessment T^rocess Model) 

For the authors there was a natural and logical sequence of events 
leading to a concern For assessment* During 1977 a plan was created and 
carried out at Syracuse University for the development of an integrated 
competency-based program for the preparation of media professionals in 
response to a mandate from the New York State Education Department. A 
description of this process has been provided in the publication, A Proc- 
ess For De veloping a C ompetency-Based Education Prog ra m for Media Pro - 

1 

fessiona ls published by KRIC in 1^78 (ED 149 7A0) . In this document, 
the techniques for identifying competencies and generating consensual 
agreement on their stated form and on their level of specificity are 
clearlv and carefully delineated. The realization that the major work 
lav nhoad in wrestling with assessment methodologies became clear as a 
rt^sult oF that project. 

A preliminary .survey of the literature on competency-based educa- 
tion outside the media field showed that the lack of clear guidelines 
for lievelopinR nSsSessment strategies was also true of all other areas 
wiii^re compctencv-bnsed education was being implemented. This led to 
the (11 V e I opmen t of .1 proposal 

to (ievel(:»p .in nssessment model that will provide a high 
degree of Flexibility for students in the most parsimonious 
manner posj^ible in order to best use the limited resources ^ 

oF the scho(^ 1 . ^ 

A series oF speoiflr (objectives were put forward and a plan to accom- 
plish their achievement was proposed. In July of 1978 this project was 
run<itvi bv the U.S. Office of Education through the Bureau of Libraries 

7 
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No 



and Learning Resoarces, and the project was begun. ^ 

The methodology for this project involved worklhg back anc^^forth be 
tween literature review and theoretical model developipent and the collec 
tion of empirical data testing various aspects of the model with formal 
and informal interviews of various experts to critique the developing 
model. A cost-effectiveness approach was selected as the most aj^pro* 
prfate method. The primary target audience for the study was the edu- 
cation community. The ultimate goal was to be able to provide educa- 
tional decision-makers with an array of assessment strategies for each 
individual competency with costs (both monetary and other) and effec- 
tiveness measures associated with each strategy so that the educational 

decision-maker could make an informed choice. The intent was to 

/ 

demonstrate trade-offs clearly, ^ 

Competency-B ased Education for School Library Media Specialists 

Competency-based education originated as an improvement of stan- 
dard teacher training programs. It has spread to other professional 
education programs, such as nursing, social work, pharmacology, and 
lihrnry/medta service. CBE as a curriculum innovation attempts to 
(ievplnp skills and attitudes which are essential for performing a set 

of spocifled tasks wl^lch have been identified as necessary for success 

3 

in a ,t?;iven profession. Three basic assumptioiis underlie CBE. First, 
fnr most jobs, it is possible to identify a finite group of observable 
b'^haviors (compet t^nc ies) which are necessary and sufficient for compe- 
tent professional performance. Second, appropriate performance meas- 
ures for each competency or group of competencies can be designed and 
validated. Third, performance assessment should take place in settings 



5 

as like actual professional settings as possible. 

In part the emergence and wide-sj^reacr'acceptatlce of CBE derives 
from a 4>erceived overbalancing In education towards the purely theo- 
reticki. Education has been noted for the continuing tension between 
tlieory and practice, between thinking and doing. A theoretical educa- 
tion deals in abstraction. The theoretician sees knowledge as an end, 
valuable for its own sake. A theoretical approach tends to be diver- 
gent, entertaining many possibilities and ideas. The nature of grad- 
uate, education leads educators to value a theoretical approach more 
his;hly than a practical one which deals in concretes, seos knowledge 
as n means to ^^n end, and tends to be convergent, focusing on Improv- 
InR proficiency in specified performances. 

Obv iously boch approaches are important if one is educating a 
professional who must be able to perform adequately on the job, but 
who also must hold a larger view of the job and its environment and 
wht^ must be prepared to adapt to unforeseen future changes. The CBE 
;3ppri)ach appears to be a response to this continuing theoty-practice 
conflict that will pull professional education closer to the "real 
world" of practice. The danger, of course, and the concorn that many 
educators have expressed, is the fear that graduate professional edu- 
cation mav become overbalanced in the practical direction and that 
theory will be abandoned. Many of the developments in competency- 
bnsod education in graduate- schools can be traced to this concern. 

''here are^ basic differences between CBE as practiced in graduate 
education as opposed to undergraduate priir.ary and secondary education. 
Most of these difteronces are a result of the need to blend the con- 
x/ergont method of CRI- with the divergent environment of graduate edu- 

10 



cation. One difference is the more general level of objective sped- 
ficity. A broad concept of competency is usually adopted for profes- 
slonal education rather than a narrow definition that spells out tasks 
in behavioral terms. The broad concept helps to^ft^cas on specific 
responsibilities which professionals will be expecteci to perforin on 
the job. The narrow definition wouxd not leave enough room for other 
learnings which are also important but difficult to specify* 

Another difference is that the model of mastery learning is not 
assuned at the graduate level. Mastery learning asserts that the ma- 
Jority of students can learn the" contents of a given curriculum If they 
are provided enough time to master the material at their own pace. 
Most professional schools adopting CBE do not accept the mastery model 
for the following reasons. First, there has already been a rigorous 
screening process designed to guarantee that those admitted to gradu- 
ate programs are capable students and can accomplish the work presented 
in the curriculum. Second, because competition for admission can be 
high, those admitted should not be allowed an indefinite period to 
master material. Finally, professionals being produced by graduate 
schools are often placed in positions of responsibility immediately, 
'^'.ms It seem^^ neither reasonable from a school's perspective nor re- 
sponsible from a broader societal view to allow students an extended 
period to reach mastery when repeated failure may indicate a lack of 
ahility or of disposition for professional practice. As a result, CBE 
In graduate school is less an attempt to deal fairly with each student 
in allowing that student to learn at his/her own pace, than an attempt 
to certify that each graduate is capable of delivering professional ser 
vice In the field upon successful completion of a course of study. 



For similar reasons (especially for the social cost involved) grad- 
uate CBE usually does not include remediation opportunities nor pjace 
much emphasis on providing multiple learning approaches. 

Finally, CBK the graduate level Is generally more idealistic 
and futuristic- The competencies of graduate training are desi'gned to 
be both skills necessary at present in the field, documeated through 
field practice, and competencies which ought to be used, but because 
of a lack of training or opportunity are not being demonstrated in cur- 
rent practice. Generally then, one can say that competencies in gradu- 
ate school are required more to protect the clients of graduat pro-- 
fesslonalf^ than to asnure students of personal well-bein^^ or fulfillment. 

These conclusions about the evolving nature of CBE in graduate pre.-- 
grams have been readied in part by an extensive study of the writings 
on CPK and throuph interviews with those working in the area. The next 
section describes the current scene. 

The St/^te of t he Art o f Competency Assessment 

Major work on CBF assessment for school library media programs goes 
on ap.ice. P:ievcn states now have competency-based certification pro- 
y,^.ims and another nineteen are in various stages of development. Of 
t.r'^u^ MarvKind (through the graduate library school), Utah (through 
tlie State Kduc^ntion Department in conjunction with several tencher 
ir. lining inst i tut lonfO , and New York (through the program at Syracuse 
rniversltv) have provided printed material of particular relevance, 
BotI) Marviand and Syracuse University work within existing curricula 
In large library scIk^oIr which are only fractionally concerned with 
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the school media program. Thus both tend to take an incremental ap- 
proach — identifying opportunities where students can attain competencies 
through general school courses, developing new courses, modifying older 
ones, and, in general , working with faculty styles of assessment. The 
State Education Department at Utah opted to support the development of 
a comprehensive pencil and paper testing program that established indi- 
cators for each competency and objective questions to assess student per- 
formance on the indicators. 

Chisholm and Ely's book remains the major work in the field. ^ The 
American Association of School Librarians on the candidate assessment 
process for library media specialists^ must also be considered a sem- 
inal work. The AASL group attempted to go beyond pencil and paper tesjts 
by providing examples of situations using jury panels of experts and 
actual performance activities. 

A number of programs outside the media field, where advances in 

assessment techniques have been achieved, were identified. A brief 

mention of more notable programs follows. There were two approaches 

to assessing teacher competency, one in Oregon critiqued by Schalock,^ 

the other in neorcjia described by Okey et al .^ Good reliability and 

g 

v'lliditv studies nccmpanied the latter effort. ' The undergraduate 
redlcnl curriculum of the Southern Illinois University School of Medi- 
cinr» has been competency-based since its inception in 1973, A good 
swmmnrv of procedures with attention to the design of a competency main- 

tenanto system and comment on eight assessment procedures is provided 

q 

by Williams. The competency-based pharmacology program at the Univer- 
sity of Minnesota employs an assessment center approach to competency 
measurement.^^ For school social workers, the University of Washington, 
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Seattle, features a consortium-directed competencv-based program fea- 
turing a candidate review board thdft uses a peer review and advocacy 

process. Kllis and Bryant descril^e some of the problems of evidence 

12 

gathering and weighing inherent in this approach. Alverno College, 

a small women's liberal arts college in Milwaukee, Wisconsin, awards 

its degrees on the basis of a CBE program using an assessment center 
13 

concept. The military have also made significant advances in assess- 
ment of competency through validation studies, behavior sampling, high- 
ly sophisticated simulation laboratories, and development of criterion- 

1 A 

referenced/performance-based measures. Especially interesting ap- 
proaches were those reported at the 1979 AECT Conference by Worth Scan- 
land of the Naval Education and Training Command, Wayne Waag of the 
Air Force Human Resources Laboratory , Robert Wilshire of the Army 

Signal Center, and Marlorie Kupp'^er from the U.S. Army Engineer School 

1 8 

in Fort Belvoir, Virginia. 

fleneral research on evaluation also provides some useful insights. 

DeProspo and Liertenrr^ describe current evaluation models in a concise 

19 

but thorouRh review for media program evaluation. Hall and Jones, 

in a general presentation of the theory and state-of-the-art of CBE 

r.iisi' and discuKs twolve provocative questions affecting competency 

assessment . "^^^ Houston and Howsam review CBE with emphasis on teacher 
2 1 

flnration," while llodgkinson et al . focus on assessment in the higher 

22 

education area. Harris and Kelly elaborate the higher education 

23 

viss(\ssment work more* Tully. Finally, two new books published within 
the l.isr year demonstrate the substantial growth in the field. On Com- 

l)etenc.e^ brings together review articles by experts on major aspects of 

24 

CRK. nf partic.tilar interest is the fine state-of-the-art review on 
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assessment by King. 

In the second recent book, Gilbert proposes a behavior engineering 
model. He suggests that behavior h*is three aspects: 

Information coming to the person telling him/her what to do 
(S^ - the discriminative stimulus) 

The person responding in some way 
(R - response) 

The action of responding and the stimulus being reinforced 
(SR - the reinforcing stimulus) 

Performance also has two aspects: a person with a repertoire for behav- 
ior (P) and a supporting environment (E) . Gilbert develops a simple 3x2 
matrix as a way of examining competent performance with its antecedents 
more analytically. He suggests a new measure — the PIP. A PIP (Potential 
for Improving Performance) is the ratio of exemplary performance to 
typical performance. Through Its use he identifies points where the 
opportunity exists to make the biggest positive change. Gilbert further 
specifies ways of measuring any performance through selected measures of 
qunUr.y (defined as accuracy, class, and/or novelty), quantity (rate, 
timeliness and/or volume), and cost (labor, material, and/or management),. 
(Gilbert's approach is an interesting extension of Skinnerian behavior 
nunUflcation and one that we would like to apply in future study of the 
l ie hi work componont of CBE. 

From the aforementioned and a number of other sources, it is possible 
to identify particular problems and approaches in the assessment of compe- 

f 

r.enoy. In the first place, many professionals speak almost synonymously 
of evaluation and measurement. Actually, there is a growing consensus 
chat evaluat^lon is the broader term relating to programs and products more 
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than to individuals, whereas assessment refers to the specific evaluation 

27 

of learnluR outcomes. Rountree's delightfully literate discussion pro- 

28 

vldes additional support i or this distinction. 

Major problems continue with respect to precise definition of meas- 
ures, determination of a ''significant" sample of behaviors when collect-- 

ing data on the measures, and, perhaps the biggest problem — validation 

29 

of the measures. Assessment centers are being established in many 
areas particularly for management development prograins. A good over- 
view on the potential for the assessment center approach is 

t2 plying the Assessment Center Method . ^ 

A number of studies used some variation of a matrix as a systematic 

organizing tool bocaurfe of the array of evaluation strategies, learning 

domain categories, pre ^enta tioi"i formats, etc. Hodgkinson says that CBE 

n^^ed.q ''instruments cnpable of much finer gradations, instruments capable 

of providing formative diagnostic advice, and instruments capable of 

31 

Ipllinc; us if the student is "real world" cor.tpetent . . . . " They see the 
murix ns a device which facilitates triangulation (that is, different 
me.isuros coming from different sources but all indicating the same re-^ 
i^ult) greatlv increasing the reliability of judgments based on the 



nsiire. Rogers u«es a matrix to relate testing strategies in the field 
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(f library science .ind media. Of course, one of the earliest to use a 
.Ti.itrix strategy to present massive amounts of interrelated information 

33 

in An onlorlv fnshion was JIMS — the Jobs in Instructional Media Study. 
The Piatrix is a device that we have adopted as useful for this study. 

lYom this comprehensive and still on-going state-of-the-art review 
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as well as from our process of empirical data collection and sorting, it 
has been possible to tievelop a clnssif ication of assessment stratoRles 
which will be descriluni In tlie next section. 

Classific ation and De finitions of Assessment Strategies 

Early in the project, specific techniques for assessing competencies 
were collected in order to establish a data bank of assessment procedures 
for each of thf* seventy competencies currently in use for the Syracuse 
proRram, plus variations and additions from other CB programs throughout 
the country. To do this a standard collection instrument had to be de- 
vised* A list of the categories used to collect the information is shown 
in Figure 2. 

The original expectation was that a comprehensive array of tech- 
niques would be relatively easily collectible. There was a plan to com- 
puterize the information for subsequent easy manipulation and retrieval 
and then to move quickly to testing the cost and effectiveness of alter- 
native techniques. Although nearly four hundred specific techniques 
were collected, it would require many more techniques from a larger 
number of sources for the original approach to be a fruitful one. 

This datn bnnk was used to test and refine the category scheme for 
■is^u^ssment strategies. From thin approach eight strategies were identi- 
fied whieh seemed to represent the most comprehensive, conventional, and 
feasible methodologies for assessment. The definitions of the eight 
assessment strategies are presented below. 
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Competency Statement: 
Competency Mumber: 
explicat!c\: 
Competency i^qe/w 
Level of Assessment: 
AssESSMEN-^ Strategy: 
Conditions : 
Criteria: 
Indicators : 
Activities: 



M 

CO 



Source: 
Access 

^U),?. - Data Collection Categories for Competency Assessment Techniques 

18 . 
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Field Experience , Field experience refers to assessment 
which occurs In the real world context, that Is, assess- 
ment which occurs In a real Job situation or internship. 
Performance which occurs in this situation Is subject to 
real world effects and tends to demonstrate competence 
at Ics most functional basis. 

Ac tual Demonstration . Actual demonstrat,ion is assessment 
which occurs in a field setting where many b ut not all of 
the real world effects are allowed to Influence student 
performance. Since student access to field settings oc- 
curs through invitation or through agreement with cooper- 
atinR institutions, some control exists to limit real 
world effects. 

Simulated Demonstration . A simulated demonstration re- 
fers to a microcosm type demonstration. In this case 
many of the variables affecting performance are con- 
trolled or manipulated. Effects due to change or real 
world contingencies play only a small part in the assess- 
ment situation. 

Portfolio . A portfolio is a collection of written mate- 
rials, audio/visual, artistic, or craft products which 
are submitted as indication of field competence. A 
portfolio is generally composed of materials developed 
during the course of a program or some professional ex- 
perience and is used to infer job related competence. 

Paper/Projects . Normally paper/projects refers to g 
product emerging from a major assignment. Examples in- 
clude term papers, slide/tape productions, musical 
composi t ions * and so forth. Paper/projects may also 
refvr to smaller assignments and projects. The idea 
here is that the work produced by the student is exam- 
ined and assessed as an indicator of some knowledge, 
attitude, or skill obtained during instruction. 

^^^^ \, .J^r>-Ail"_^A^ ' Oral presentations are student- 
controlled summaries and discussions which reveal (1) 
the activities involved in, (2) the progress of, and 
(3) tho findings related to student pro:jects or papers. 
The oral presentations are more than impromptu conver- 
sations. They allow the audience to ask questions or 
make comments which may require the extension of ideas 
through in depth responses. Thus, such testing is 
more Flexible than written presentations, though less 
detn I led , 

fl^/lLJlLlt- ^^''^1 t^st is a strategy in which stu- 
dents respond to a set of questions specified by an 
examiner or Rroup of examiners. This strategy allows 
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flexibility in pursuing answers more fully and clarify- 
ing responses, though this is generally at the expense 
of the organization and detail present in a written 
response. 

(8) jjritt£n Test . Written tests are generally of two types: 
selected response (true-false, multiple choice), and 
constructed response (short answer, essays). There is 
virtually no interpersonal interaction in such tests, 
and competence is assessed on the one-way communication 
of the student with the written materials. 

A separation into two subdivisions of (a) measures of competence and 

indic ators of competence, as illustrated below, seemed most appropriate: 

(a) (1) Field Kxperience ) 

(2) Actual Demonstration ) Measures of 

(3) Simulated Demonstration ) Competencies 

(b) (4) Portfolio ) 

(5) Paper/Projects-N. ) ^ 

(6) Oral Presenfr^tioh ) Indicators of 
, (7) Oral Tests ^ ^ ) Competencies 

(8) Written Tests ) 

Assessment stratGgieB as measures are distinguished from strategies 
as Indicators because measures assess actual performance in real or sim- 
ul.-ued iob settings, while indicators assess behaviors thought to be more 
or loss rolated to job performance. The assessment strategies of (1) 
field expLTienct!, (1) actual demonstration, and (3) simulated demonstra- 
tion are said to measure competence because they attempt to record the 
existence of specified, necessary job skills and attitudes (i.e., the 
identified competencies). The assessment strategies of (4) portfolio, 

p.iper/projects, (6) oral presentations, (7) oral tests, and (8) 
written tests are said to be indica tors of competencies because they do 
not lest .ictual performance either in simulated or actual job situations. 
Instead, from these stratcpies, one infers what performance might be like 
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In these nltuations. 

This highly useful dlHtlnction between measures and indicators led 
to the problem of determining "good" techniques for assessing eacjh com- 
petency. 

'1? ^ res oF E ffectiveness 

Effective assessment outcomes are critical to a competency-based 
program, Petermining the effectiveness of CBFw assessment outcomes has 
been a problem in the CBE movement since its inception. Many of the 
problems may be due to the limitations of. the art of measurement. These 
problems, however, may be more a matter of lack of experience with the 
technology of performance assessment than a true limitation. 

Our approach to effectiveness analysis was first to ask the ques- 
tion: "To what degree do the assessment strategies accomplish what they 
set out to accomplish?" This is more or less equivalent to asking: Is 
the assessment strategy v alid ? Thus, in one sense, effectiveness analy- 
sis of assessment strategies will be identical to a validity analysis. 
This technique was selected as the most appropriate approach to perform- 
ing n primary of f ect- i veneas a nalysis . 

Validity annlysis seems a necessary but not sufficient response to 
devb^li^pii^^i; mensuros of effectiveness. A number of other questions are 
nls(^ Important (n considering the effectiveness of a given strategy. For 
excimple, how practical is It? Is there a reasonable expectation that the 
reduislte mix of faculty experience and appropriate environment can be 
crtMtrd? How complex are the directions for an assessment instrument? 
If tho procedure is too complicated, will it affect the results? How 
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much effort is involved in constructing, administering, and scoring the 
assessment technique? While some people feel confident *that prJmary 
effectiveness analysis can be made to yield good quantitative data, other 
kinds? of data are aleso important in making judgments about hov "good" a 
pa>"ticular technique may be when used to assess a student's competency. 
Data on these qualitative questions are considered when performing a 
second ary effectiveness analysis • ^ 

The careful separation of quantifiable measures from qualitative 
measures clarifies the decision-making process without suggesting that 
one tvpe of measure is any more important than another. Figure 3 demon- 
strates how the evidence on the comparative effectiveness for a set of 
sttate^les for each competency as well as for clusters of competencies 
can be arrayed in matrix form. What follows is a brief exploration of 
procedures for primary and secondary effectiveness analysis. 

I^ rimary Effectiveness Analysis , A basic assumption of CBE is that 
assessment strategies which examine performance provide the best meas- 
ures of competence. The closer measures are to real world contexts, 
the more valid they tend to be. Strategies which are increasingly re- 
moved from real world contexts are potentially less valid as meas ures of 
competence, Bv this reasoning a simulated demonstration would be less 
valici Chan a field-based assessment (by how much is not known). Simi- 
larly a portfolio asf'essment would be less valid than a simulated demon- 
stration. Validity Is defined as the extent to which an assessment 
strategy does the job tor w!iich it is used. ^^en a strategy which has 
been dt-»i^',ned to rmjism'e a competency actually does so, It Is said to bo 
valid. Wht^n an assessment strategy which has only the power to infer a 
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competency is used to measure one, there is need to question its validity. 

Effectiveness analysis of any assessment strategy is based upon one 
fundamental question: To what degree does the given assessment d8 what 
i t is intended to do ? Attempts to answer this question have been a pri- 
mary focus for those in the field of educational and psychological tests 
and measurements. The research literature from this field can be of 
considerable assistance in resolving competency-based assessment problems. 
Holloway strongly states that the future success of CBE problems will de- 
pend upon what CBE educators know about the general field of tests and 
measurements . 

It is the concern of experts from the tests and measurement field 
to mons'urc qualities, characteristics, skills, and/or competencies within 
an individual and to determine the effectiveness of their measures. The 
term "test" is usually considered to mean the presentation of a standard 
s^t of questions to be answered, the result of which is a score or nu^ 
moricMl value which is said to represent the extent of a characteristic 
or a quality that a person possesses. 

Test developers nre not always satisfied that the tests they con- 
r.truot actually measure the characteristics they were designed to meas- 
ure. This mav be why they take great pains to determine the degree to 
♦•hich thv tests aetiially measure what they are intended to measure. 
This iH rnlled test validation and requires an examination of the reli- 
iKilltv of a test plus a closer examination of various types of test 
va 1 i d i tv • 

A test's reliability is that portion of the result (the score) that 

m 

is due to svstemritic errors and persists, therefore, from sample to sam- 

3 6 

pie, or from one administration of the test to another. Keliabilitv 
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reflects the doRree to which the test can deliver the same or similar 
scores with each testing, or deliver scores which reflect the true dif-- 
ferences among students. Indications of reliablUty are presented In , 
the form of coefficien ts of, reliability > Reliability -is ccr.isidered a 
necessary condition for test validity but not a sufficiepl. one. 

The determination of whether or not a ^est measures what it is 
supposed to be measuring is examined in validity analysis. . Validation 

is a procedure that provides "evidence that an inquiry is free from 

37 

bias or otherwise conforms to its declared purpose. In the absence 
of bias, we know that the results are not influenced by other factors 
and can be believed, 

V 

Test validity is not easily attained and at best is a matter of 
degree. Mehrens and Lehmann state: 

A test may have many validities, each dependent upon the 
specific purport* for which one uses the .test. Eventually 
the validity of any test is dependent upon how it is used 
in the local situation. 

Test validation involves examination of four types of evidence: (1) 
conttMit, (2) montnl constructs, (3) relevant criteria, and (4) appearance 
F.arh of these Forms of evidence is discussed briefly below in the manner 
t vp I CM 1 1 V cmp h^yed hy test and measurement experts . 

Content Val idity , The test is examined to determine 
how the content of the test samples the domain about 
which inferences are made. An individual's test score 
^ can Vc'ilidly infer his/her knowledge of a subject only 
to the degree that the test has adequately sampled 
(i.e., provided representative examples of) the sub- 
ject domain . 

^'^^ Construct Validity. The test is examined in terms of 
the deRree Co which the test scores can be accounted 
for by certain constructs of a psychological theory. 
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It is interesting to reflect on how this notion 
might be replaced in CBE by competency validity 
which woald examine the test in terms of the 'degree 
to which the scores can be accounted for by certain 
competencies. 



Criterion-Related Validity , The test is examined in 
terms of/its relationship with external measures 
(which measure the same or similar things). For ex- 
ample, success in college (Grade Point Average) is 
indicated by certain other tests or jneasures obtained 
in high school (i.e», SAT, High School GPA) . Such an 
examination speaks essentially to predictive validity. 
That is, the ability of a test score to predict other 
types of performance. 



(^) Face Validity . Her'l|p:he test is examined in terms of 
its ability to show how "on the face of it" that it 
n>easures what it says it measures . * 



The technology of test validation is a sophisticated one, based on 
a long history of theoretical and applied research. It is possible that 
this rich ^body of research can be mined to suggest ways in which test 
validation can be applied to forms of assessment other than the written 
test. "This Is the next major phase for research on competency assessment 



f ^Gco ndarj^ Effectiveness Analysis . A number of qualitative dimen- 

' sions have been proposed In the literature that also purport to measure 
thf effectiveness of assessment techniques. Examples of these dimensions 
Inrlude the determination of adequacy and appropriateness of assessmer" 
T)ULCOmes. Doughty et al. suggest communicability , feasibility , and 
utility > Knapp and Sharon add equitabillty . Kelly proffers ethi - 
caj^lty. Meherens and Lehmann propose objectivity and p racticality 
Cost, (if course, Is a major criterion, so much so that it has been sep- 
arated from theoifipr dimensions and treated as an independent category 
to be discussecf in the next section. 
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Each of these characteristics pertains to some dimension of an 
assessment strategy which may influence the choice to keep or remove 
the strategy from the tot^l assessment progfam. For example, equita - 
^iiilX addresses the fairness of the assessment strategy. Scriven advo- 
cates an '"inspecting process" which searches for injustices within the 
assessment strategy. 

Whatever the outcomes, there are certain types of proce- 
dures (sic) that are inappropriate for moral reasons, and 
inspection of process must be made to see whether exc'ess 
* cruelty, inequitable methods of grading, etc., are impor- 
tant.** ^ 

Clearly where there is evidence that the assessment technique is really 
not fair to the student, educators are obliged to discard it, or to 
alter it until it is equitable. 

Two problems are associated with secondary effectiveness analysis. 
One is the problem of what to consider when conducting secondary analysis 
Since there are many characteristics suggested and since some even appear 
to conflict, selecting the appropriate ones can be a problem. However, 
trii$5 perhaps should be treated as a local problem to be decided according 
tc the circumstances and conditions under which the analysis is taking 
place . 

The second problem relates to definition- Most of the terms sug- 
gested above are too vague to apply reasonably. Even though the ulti- 
mate meaning of any of the terms will be ^.etermined by the location and 
conditions of analysis, a more precise and common set of definitions 
would be an important contribution. This also is part of the next phase 
of research. 

Secondary effectiveness, however poorly defined at present, is 
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Important because it imposes a realistic perspective on ef f ectivenessr 
based decisions. There are always more things likely to effect the 
selection or rejection of an assessment's strategy than validity alone. 
Secondary effectiveness analysis assures a more comprehensive analysis. 

A distinction Is often drawn between measures of effectiveness and 
measures of efficiency. Efficiency questions focus on the consumption 
of time, money, and effort. Those concerns are addressed in the next 
section on cost analysis. 

Me asures of F^fficlency — Cost Analysis 

In general, cost analysis attempts to discover the potential and/or 
actual costs of a program. As the term is usea here, cost means the ex- 
penditures of human and monetary resources for the development and oper- 
ation of programs. As it Is currently conducted, cost analysis concen- 
trates largely on the monetary resource expenditures. However, there is 
increasing attention given to the expenditure of human resources as well. 
This distinction between human and monetary costs provides the basis for 
a two-part analysis of costs in much the same way as that drawn above be- 
tween primary and secondary effectiveness analysis. Monetary costs are 
quantifiable and will be used for the primary cost analysis. Human costs 
nay be more qualitative in nature and will be the focus for secondary 
c .vst ana lysis. 

Prim ary Cost Ana lysis > Cost, as defined by economists, is "the, best 
alternative forgone.'* However, cost will be used here in a more gener- 
al sense as "a common and universal measure of the nature and quantities 
.^f resources used in a specific manner to achieve stated objec tives . ^ 
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A universal standard is necessary for the comparison of alternatives 
(proRrams, assessment strategies, etc.) in order to choose among them. 
In cost analysis, the universal standard is usually dollars, and although 
there are some difficulties that will be discussed below, it seens the 
best reflection of cost available. 



Life-Cycle Costing 
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Fig. 4. Life-Cycle Costs 
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We have adopted n function by resource approach to costing followlnR 
Doughty and Bellbv, Carpenter, Stace, and others. Program activi- 
tlort (functions) are analyzed In terms of the expenses they generate. 
Tht:se expenses are then distributed among the various categories of re- 
sources. The cost of these resources (personnel, equipment, facilities, 

material) is then estimated using standard prices for comparable re- 
f 

sources. Methods of costing not based on the resources necessary for 
each Individual or for each specific program or activity can suffer from 
ono of two possible errors in estimation. ' First, costs may be included 
which would not be charged to any single program, because, for instance, 
they are shared among several programs. These are referred to as joint 
cosjts and will be discussed more fully later. Including joint costs en- 
tir<?ly In a single program or activity thus inflates the apparent costs. 
The second error is the obverse of the first. Costs which are obscured 
by more traditional accounting and budgeting procedures may be omitted 
thiis depressing the actual cost of a program. "(Function) /Resource 
cost analvsis is designed to provide cost estimates which permit equita- 
ble comparisons among projects (strategies), where cost or cost/effec- 
tiven(\ss Ls tie b--Jsls of comparison. " 

52 

Although the cost model used by Doughty et al . and others exam- 
ituHl developmental as well as operational costs of programs, we will 



disrejL>tird the developmental costs and concentrate on operational costs. 
The shaded portion on I'iRure h indicates our Focus. This Is done in 
order to nnalvze costs along a third dimension — that of the eight assess- 
ment strategies. The cost analysis model, then, analogously to the 
effectiveness model, ran be shown as a three dimensional matrix display- 
ing operational costs generated by the interaction of functions with 
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resources with aaseasment strategies » Figuie 5 graphically portrays 
those dimensions and their interrelationships. 

The procedure begins with the identification of a set of functions 
that are mutually exclusive yet collectively include all the activities 
th.1t take place In the assessment procr^ss. A parallel process involves 
th^^- determination of all the resources used. Both function and resources 
categories must be carefully and explicitly defined so that an observer 
or a participant can tell what is included in a specific function and/or 
to which categories a certain resource belongs. The next two subsections 
provide the function and resources classifications with definitions. 

Ass€^ ?4sment Funct ion, Assessment functions are the discrete activi- 
ties associated with an assessment strategy. Functions represent what an 
Lristructor, supervisor, or credent ialing agent actually does in the course 
of assessing the student. Six assessment functions have been identified: 



(I) Development or Kovision (of instruments and procedures) 

(?) AdminiKtrat ion 

(J) Analysis 

(4) Management 

(5) Advisement on Non-Competence 

(ft) Aggregate Judgment of Competence 



Thi\S(:» are defined as follows: 



(I) nevelop ment/Revision . After the initial development of 
Ins^trumrnts and procedures, revision refers to the ad- 
justments made of an assessment strategy prior to its 
use. For example, the questions on a test may need re- 
vision. The instructor may improve questions, test 
procedures, nr format. Revision also includes the in- 
corporation of new information. In fields where high 
technology or major philosophical shifts require fre- 
quent, drastic revisions, this function becomes a cen- 
tral cost. 
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(2) Administration . Administration includes the time and 
procedures necessary to assess students. Traditionally, 
the standards for this function are the classroom time 
and activities used to give paper and pencil tests. 
Administration becomes more complex and costly as higher 
order assessment strategies are employed. Included in 
Administration are travel costs for site visits. 

Analysis . Analysis refers to the activities associated 
with examining, processing, and Interpreting performance 
data from a particular strategy. Judgments of competence 
or non-competence are based on analysis. The most famil- 
iar form of assessment analysis is test corrf.ction and 
scoring. Less familiar forms of analysis ate (a) observa- 
tions related to field assessment, and (b) analysis of 
paper/projectsi, portfolios, and oral presentations. 

* 

i^) Management . The management function refers to the activ- 
ities of recording and reporting data and analyses. Nor- 
mally these activities relate to student grades. However, 
where instructors approach assessment data with an eye to 
developing or improving student competence, recording and 
reporting assessment data become crucial to providing 
valid diagnosis and prescription. Management also in- 
cludes planning assessment programs for a semester. 

(5) Advisement on Non-Competence . Advisement on non-competence 
involves all the activities of dealing with students wha 
fail to achieve necessary levels of performance. CBE pro- 
grams normally recycle these students, but only after ad- 
vising them of error and misconception. In graduate level 
courses this function may lead to counselling students out 
of a program or field and, if appropriate, into another. 

Aggregate Judgment of Competence . This function refers to 
the summative judgment given on a student's cumulative per- 
formance at the conclusion of a program. It is the point 
at which the department, institution, or credentialing 
agency certifies the overall student competence. The func- 
tion, as Its name suggests, is the aggregation of all the 
separate assessments into some single, final decision that 
a student is or is not ready for professional practice. 



'l^t)U£ces. As proponents of program budgeting know, identifying all 
the resources used solely for a particular program, (or, in this case, an 
assessment strategy) is not an easy matter. Institutional budgeting and 
nocounting systems are generally based on jurisdiction rather than function/ 
resource. Jurisdictional accounting, as the name implies, accounts for 
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the allocation and expenditure of funds by each organizational unit 
rather than for the actual function for which the funds are used. Juris- 
dictional, or llne<-ltem, accounting is too general and undifferentiated 
to use in attempting to determine how much a given program or set of 
assessment procedures will cost. 

In order to Identify the resources which go into a specific ptngram, 
it is necessary to identify the various functions and then to specify 
what Croats are constituted by resources used in these functions. Cost- 
ing generally occurs, then, by estimating the percent of time for facul- 
ty and staff, the percent of use for facilities and eqaipment (where not 
totally allocated to a single program), and the cost of consumables. 
(Typically for education, the major costs are for faculty time.) These 
functional estimates are then translated into dollar costs, where possi- 
ble, from accounting or budget figures. 
» 

Resources, then» are the entire variety of individuals and mater- 
ials (people, supplies, equipment, and facilities) necessary and avail- 
able for carrying out the assessment functions. Five resource categories 
are identified for cost analysis of assessment strategies. 

(1) Personnel 

(2) Equipment 

(3) Facilities 

(4) Support Services 

(5) Consumables 

They are identified as follows: 

(1) Personnel . Personnel are those individuals who function 
within a discreet instructional unit such as an academic 
department, or in association with such a unit. Typical- 
ly they are (1) instructors and field supervisors, (2) 
managers ♦ and (3) support staff. The critical group con- 
sists of the Instructors and field supervisors who handle 
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the actual student assessment. Therefore, an anal- 
ysis of how Instructors and field supervisors spend 
their time on student assessment is critical to a cost 
analysis of that activity. Howeveri the contribution 
made to the assessment process by unit managers (e.g., 
chairpersons or deans) and secretarial and clerical 
staff must not be overlooked. Thorough analysis will 
consider the total contribution of all personnel with 
respect to the assessment process. 

(2) Equipment . Equipment Includes the durable, reusable 
hardware and software used in the assessment process* 
These may be books, computers, games, media equipment, 
etc. Equipment Is distinguished from consumables be- 
cause the former Is used more than once. 

(3) Facilities , Facilities are classroom, office, and 
other specialized areas necessary for assessment. \^en 
a facility is basic to all assessment strategies, it is 
not usually Included in cost analysis. However, where 
a specific facility is necessary to only a certain type 
or class of strategies, it musf be costed. 

(4) Support Services , Support services refer to those serv- 
ices provided to aid the assessment process. Examples 
are (1) teaching assistants whose duties consist of ad- 
ministering and analyzing student assessment data, and 
(2) computer services used to analyze and report assess- 
ment data. Important new support services in assessment 
programs are those provided by assessment centers. 

Consumables . Consumables are those materials which are 
used during the assessment activity, but which are not 
re-usable at its conclusion. Typically these include 
answer sheets for paper/pencil tests, record keeping 
booklets, computer printouts used in assessment, obser- 
vation forms , etc . 



Primary cost analysis thus Involves the direct costing of function times 
resources tlmos strategies and the eventual derivation of an actual dol- 
lar amount for each assessment strategy or combination of strategies in 
a Riven assessment system. A cursory inspection of the cost analysis 
mntrix (Figure 5) may make the costing procedure appear deceptively 
easy. However, there are many problems, major and minor, that should be 
acknowledged in conducting cost analyses. Three cost-related concerns 
fundamental to the proper application of the model described here are: 
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(1) Distinguishing different types of costs 

(2) Choosing the basis for cost comparison 

(3) Accounting for variance in time and usage estimates 

A brief description of each of these follows. 

Types of costs . It is essential to be able to distinguish among 
different cost categories as confusion nf one type of cost with another 
will distort the analysis and result in either under- or over-estimation 
of costs. Some of the basic costs which a cost analyst must be able to 
differentiate are fixed and variable costs, recurring and non-recurring 
costs, joint costs, and sunk costs. # 



Fixed costs are those costs which remain stable despite fluctua- 
tions in h program's operations. Most major capital expenses are re- 
garded TiS fixed costs. Costs for facilities are usually fixed, since 



bulldipg nortgapjes and upkeep continue at a fixed level whether there 
are many or few students. Variable costs are costs which are directly 
related to the level of activity of the system. Thus, expenditures for 
m/.terials are variable costs, since the more students there are, the more 
materials are required. 

Recurring costs are those which occur during each and every cycle of 
a program's operation. Non-recurf ing costs are one-time costs during a 
program's life; for instance, costs for research and development. If 
these two types of costs are confused, especially when a non-recurring 
cost is listed as a recurring one, the entire program operation cost is 
inflated. Tn the cost matrix presented for use here, there are only re- 
curring costs, since the analysis deals solely with operational costs of 
assessment systems. 
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Joint costs are those costs which must be allocated to more than 
ont; program or jurisdiction. They are costs which are shared. For ex- 
ample, different instructional programs may share the same equipment, 
or teachers may teach in n^ore than one program. Determining percent- 
ages of use and time for such joint costs is difficult but crucial. 

Sunk costs are those expenditures which have already been made at 
sone timf^ in the past. A common error in cost analysis and in decision- 
milking is to include sunk costs in future projections of the cost of a 
program. Once resourc,es have been irretrievably allocated, they should 
be disregarded in calculating program costs. This reflects the approach 
of using the "piesent worth" concept of. economic analysis, which allows 
all costs and decisions to be based on the value or resources at the 
mc^ment of analysis. The general principle behind ignoring sunk costs is 
to avoid committing resources which are still flexible in the present, on 
the basis of the past ^rth of resources which have been allocated and 
therefore, in effect, no longer exist. This procedure allows each de- 
cision to be based solely on the current value of flexible, available 
rfps'ources. 

Basis of Cost Comparison . There are essentially two kinds of com- 
parisons which a decision-maker may want to investigate. First, deci- 
F ion--:nakors may want to compare the absolute costs of various strategies 
or clusters of strategies in order to identify the general array" of 
costH Involved In every strategy or combination of interest, the focus 
Ifi on general stratep,y comparison for which strategy comparable costs 
must be gathered These costs include virtually everything which is 
necessary to any of the assessment strategies. There is one exception. 
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Where all strategies require ^he same basic resource (e.g., classroom 
space), then that resource need not be Included since adding this cost 
merely Inflates the cost of each strategy by a similar amount and gives 
an unduly high esti^nate for all strategies. Resources which can gen- 
erally be eliminated from costing in this manner are basic resources 
which are fundamental to the' enterprise of education and not unique to 

r i 

any particular program or activity. 

The second type of comparison that decision-makers may want to in- 
vestigate is one which indicates which .strategy or groi/{) of strategies 
is most appropriate for a particular situation or setting. Program 
«^JL^Ii5_._cost_s.are appropriate for such comparisons. These costs are 
costs only for resources which are not already available to a program. 
Resources which a program already has would not be included In costing 
of a particular strategy. For instance, if a simulation strategy re- 
quired use of an assessment center, and a center already existed and 
was accessible, then construction, maintenance, and staffing would not 
need to be included. Only the tosts per use of the center would be 
figured into the strategy. However, if no such center existed, then all 
the rosts associated with creating one would have to be included in the 
strategy cost analysis. 

These two different comparison contexts— strategy comparable costs 
and program' specif ic cosr.s--are both Important depending on the level of 
decison-making involved. If a federal or state agency is estimating 
average costs for a series of assessment systems, the -former method may 
be more revealing in providing some sense of overall costs. If a spe- 
cific program is attempting to determine its own system, the latter anal- 
ysis is, no doubt, of 'more use. 
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T ime and Usage Estimation * A major problem Is the lack of validity 
and reliabilltv data for the various estimates of time and usage which 
underlie the cost in the function/resource matrix for each strategy. How- 
ever, there are ways to Increase the validity and reliability of the esti- 
mates. First, clear definitions of functions will help the estimator (and 
the faculty member) distinguish aw .ivlties (functions) and so estimate 
mofi? accurately the proportion of time spent on each one. Second, redun- 
dancy of estimation in the form of multiple measures repeated over time 
will help identify any inconsistencies and provide some sense of the 
variance in responses. Third, well-conceived and validated measuring in- 
struments and techniques will obviously Increase confidence in the final 
est Imates. 

Rating scales, interviews, self-report plans, and institutional re- 
quirements are some of the typical procedures for gaining such Information. 
Moro sophisticated techniques, for example, some of the procedures used 
in perceptual psyciiology, decision analysis, values clarification, and 
research, can be adapted when the costs are significant enough to warrant 
su.!h an investment. / 

In the last analysis, a sensitivity analysis may reveal that over- 
or under-estlmates of time may vary widely before any substantial effect 
on the choice of strategy occurs. For exaraple, •it is quite possible that 
anv systematic over- or under-estimatlon of time or usage may occur across 
all strategies and thus affect all estimates equally. In sum, where the 
analysis seems grossly distorted, then experience, common sense, and in- 
taitlon should be used to help give a more accurate reflection of true 

C-Ofi ts . 

41 



35 



1 



. Secondary Co8t Analysis . By and. large, current cost analysis disre- 
gards non-material costs because it is difficult to assign numerical 
values to these costs. Secondary cost analysis, however, consciously 
considers these non-material costs. Typically these are social costs, 
^ psychological costs, lost opportunity costs, and the like. Because 
dollar values are not easily assigned to such costs, they tend to be 
overlooked and thus important .decision-mal^ng information is discounted. 

In fact, the designation "secondary cost analysis" is a bit mis- 
leading. The consideration of non-quantifiable costs and unintended 
outcomes is secondary only in the sense that the techniques for evaluat- 
ing such costs and events are much less exact and less easy to apply. 
The concerns In secondary analysis are often more significant and have 
a greater impact on decisions than those dollar concerns considered in 
the primary cost analysis. This is particularly true when secondary 
analysis involves political and social issues which have ramifications 
for the public interest beyond the scope of dollars spent. For ex- 
ample, if a new competency-based program with high standdlrds^produces 
L^yjL^, graduates, albeit with better skills, there will be ramifications. 
The^re is a cost to the segment of the public which might have received 
some service, though of lesser quality, but must now do without any 
s^^rvlcG while waiting for better trained graduates. On the other hand, 
tht^re aro a different set of costs when more graduates are produced but 
some are only capable of substandard service. 

Some of the more fundamental secondary (or qualitative) costs for 
CAK are; 
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(I) Social costrt 

.(2) Opportiinltv costs 
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(3) Accuracy and information costs 

(4) Inflexibility costs 

(5) Political costs 

These will be discussed in turn below. 

Soc ial costs are those negative external outcomes which affect large 
segments of the pviblic. Pollution, traffic congestion, and nuclear con- 
tamination exemplify social costs. Economists often refer to these as 
''externalities," a term used to designate costs of production not di- 
rectly borne by the producer. Externalities may produce either positive 
or negative effects. 

Opportunity costs are the cost of the next best alternative for 
which resources could have been used. This definition coincides with 
the economic definition of "coat" referred to earlier. It simply means 
that if the alternatives are to select a performance-based CBE assess- 
ment system or a new information management program, the cost of the 
assessment system is conceived as the cost of the new program, the next 
best alternative given up or foregone. Expressing costs as opportunity 
losses is often much more dramatic than simply presenting a dollar ex- 
penditure amount because It forces people to confront graphically the 
things they are giving up wKsn they choose one alternative over another. 

Acc uracy and information costs refer to the cost of acquiring addi- 
tional information in order to be able to make a more accurate* Judgment, 
Ir.creased accuracy requires Increased information which entails increased 
C ' 0 s t . 

Inflexibility costs are those costs associated with decisions whioh 
cannot be easily reversed. Often resources must be committed irretriev- 
ably in choosing one program over another or one approach to strategy 
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assessment over another and not much can be salvaged if a later decision 
is made to abandon the project. 



Finally » political coats are those costs which occur generally to 
decision-^makers, and which involve losses in power, status, position, or 
office. The most obvious case of such costs is one where a politician 
supports an unpopular law and is not reelected because of it. This is 
not a trivial factor inasmuch as much of the drive behind CBE is political 



rather than (or perhaps as well as) educational. Thus the political costs 
of adopting or not adopting competency programs and assessments should not 
be Ignored in the activity of cost analysis. 

Each of these types of cost resists easy quantitative analysis which 
would permit an assignment of a dollar value to them. Therefore, they • 
aru oftea overlooked or sidestepped. One tactic in cost-effectiveness 
studies Is to view such costs as "negative" effectiveness and to let the 
effectiveness analysts wrestle with the difficulties of measurement. Our 
position is that the secondary costs are too important to be neglected 
aivl thus hospitality to these qualitative factors has been provided in 
our mode 1 . 

Kven though dealing with these secondary costs is difficult, we can 
«n^\;e.st some >»uldolines. First, when dealing with costs which are not 
LMsilv quantified — for instance, the costs* to students of faculty evalua- 
tion whirh mistakenly eliminates some competent individuals from a pro- 
gram versus tho costs to society of faculty evaluation which passes non- 
competent students into public practice — the various costs can at least 
ho ranked according to their perceived Importance. Thus, given limited 
resources, If it Is felt that society's well-being has relatively more 
value than that of each student, then a system can be designed which 
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requires more expert performance and is leas lenient In borderline cases 
evdn though no dollar cost Is placed on such values. (In practice, dol«* 
lar amounts can oftbfi be found for such costs through the ^measurement of 
expected utilities of decision-makers or of certain segments of society. 
See the decision analysis literature for a more complete account of these 
techniques I ) 

A second technique for dealing with secondary costs la to get some 
estimates of how important these factors are in the overall decisions 
to be made. * In the above example, one approach might be to determine how 
rnwch a possible student or public lawsuit might cost an institution if it 
were determined that the assessment strategies for determining competence 
wtre either too stringent or too lenient, and what the possibilities of 
such Judgments might be. Given these estimates, an analysis could be 
mt\de which would reveal whether different strategies or standards should 
be used, and at what: point additional costs for better assessment cease 
to balanc-e thfj costs to students and society. In other words, if a stu- 
dent lawsuit might cost an institution $10,000, and the likelihood of 
such a suit was 1 in 500, then the expected cost would be $10,000 x .002 - 
$20 per 500 students trained. If the cost of assessment to reduce the 
llkellhoojj of such an occurrence to 1 in 1000, or .001, was $20 per stu- 
dent, or $10,000 for 500 students, then investment would not be reasonable. 

Manv people are extremely uncomfortable putting questions of value 
Into monetary terms, however vague. Still it is essential to realize 
that putting dollar costs on values such as "fairness to students" or 
"responsibility to society" occurs at every Junction of the academic 
process where allocations of scarce funds are made to one type of activ- 
ity and not another. The argument above is not meant to indicate that 
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student concerns are unimportant or always of lowest priority. Instead, 
It emphasizes rather dramatically that tradeoffs are funlamental to most 
derisions, and It forces decision-makers to understand that the abstract 
Idea of assessing competence has very real and often contradictory conse- 
quences for students and the public. 

Although these secondary 'costs should not be ignored, it is generally 
not worth the investment to attempt to be too refined in placing monetary 
values on all costs or anticipating fully all outcomes. More often than 
not, an awareness of these costs and potential effects may be sufficient. 

The two primary dimeftsions of the model — cost and effectiveness — 
having been discussed in some detail, the components can now be brought 
together in a discussion of the model for cost-effectiveness analysis nf 
assessment strategies for competency-based education. 

The C ost-Effectiveness Model 

Figure 6 provides a graphic overview of the model. It is drawn to 
illustrate the way in which the cost-effectiveness analysis of assessment 
programs Is to be approached. It. is once a cost analysis and again an 
e'f pctiveness analysis. The two analyses are undertaken for the purpose 
of exposing differences between alternative assessment strategies. The 
d.ita developed are then recapitulated In summary form in such a fashion 
that the decisl on-THflker can focus on the pertinent find relevant facts in 
order to make Informed decisions and reasonable policies. 

The bulk of the analysis effort is concentrated in the center of the 
model and is represented by the four blocks: 
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Fig. 6. General Cost-Effectiveness Analysis Model for COE Assessment Programs. 
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(1) Primary cost analysis 

(2) Secondary cost analysis 

(3) Primary effectiveness analysis 

(4) Secondary effectiveness analysis 

The results of the first pair of analyses are placed in the Cost Data 
Mijtrix and those of the second pair in the Effectiveness Data Matrix, 
The data from these matrices are condensed again and brought together in 
the Data Display for Decision-Making, 

The dichotomous approach — cost and then effectiveness — is used for 
the sake of completeness. It has the advantage of pointing up to analysts 
what they typically overlook in approaching cost-effectiveness analysis. 
The division into primary and secondary parts, more or less corresponding 
to a division into quantitative and qualitative measures, gives the 
decision-maker an opportunity to see what is critical and what is less so. 
The .final choice then should be based on a holistic view of the situation^ 
even though c Ircumstances. may dictate against detailed secondary forms of 
analysis. Here, however, the decision-mak6r can control the analysis proc- 
ess bv selecting which elements of secondary analysis will stay in the 
final array and which will be omitted. The detailed description of the 
elements suggested for inclusion in the secondary analyses (griven in the 
two previous sections above) should assist in making analysts and decision- 
makers fully aware of the reasons why certain secondary analysis has not 
occurred . 

The^ Matrices > Cost analysis and effectiveness analysis are greatly 
facilitated by the use of matrices. Doughty and Stakenas make a strong 
recommendation for their use in reporting and relating Information for 
decision-makers. In addition to their value in communicating informa- 
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tlon, mntrlrus are uHt-ful as a format in which to awange data en route 
through the analysis. Often matrices become part and parcel of the anal^ 
yals process and may even serve to guide the analysis Itself. Matrices 
are used In the present model for all these reasons. 

■Hie Cost Data Matilx Is used for three reasons: (1) to guide cost 
analysis, (2) to display and relate Informa Lion for decision-makers, and 
(:) as a residual for the placement of available data. For the present 
the Cost Data Matrix is confined largely to primary cost analysis. The 
growth and development of the cost analysis of competency-based assess- 
ment programs may expand the use of this matrix to secondary cost analy- 
sis as well. The reader Is referred to Figure 5 for details on the Cost 
Data Matrix. 

The Effectiveness Data Matrix Is used In two ways: (1) as a resid- 
ual for effectiveness data, and (2) as a reporting device. Principally 
the Effectiveness Data Matrix serves to display effectiveness data for 
analysis and Interpretation. Unlike the cost matrix, however, the ef- 
fectiveness matrix Is not confined to primary analysis. Still, the de- 
gree to which it will Include secondary forms of effectiveness analysis 
data is highly situational. Because an effectiveness analysis Is es- 
sentially a validity analysis, the data from such an analysis may not re- 
quire a matrix. The Effectiveness Data Matrix is recommended, neverthe- 
less, because Us use indicates to the -valuator the breadth of the anal- 
ysis and It forces the evaluator to ask secondary analysis questions. 
FiRure 3 provides the detail on the Effectiveness Data Matrix. 

.Dispjav f o r Dec Is ion-Making . The final major element in the 
penoral model Is the Data Display for Decision-Making. For the display 
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of analysis data, Doughty and Stakenas suggest a tabular array approach. 
Their claim Is that su^h an approach 
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enables comparisons to be made of different kinds of 
quantitative data as well as qualitative criteria. 
Single dollar cost measures pf an alternative are as 
uncommunicative and uninf ormative as any contrived 
single measure of effectiveness.-'^ 



They emphasize the need to let the array "reflect a set of cost and 
effectiveness measures suitable for use in making decisions." They 
further suggest a two-dimensional fieli". for comparison, which In this 
esse would be S trategy by comparative Cost Effectiveness criteria, or 



Assessment Strategy x Cost-Effectiveness Data 



Figure 7 illustrates this arrangement. 

Fixin g Costs or Fixing Effectiveness . In using the model it is 
probably necessary to "fix" at least one of the two standards of compar- 

i 

Ison: cost or effectiveness. Fixing costs or effectiveness means setting 
some minimally acceptable level of effectiveness, or some maximally ac- 
coptnblc level of cost, and allowing the unfixed factor to vary. In com- 
paring systems of assessment strategies, if there Is a fixed resource 
limit beyond which costs cannot range, then the effectiveness of systems 
whose coi^ts are under this ceiling is the criterion on which a selection 
win be made. Conversely, If a certain level of achievement Is essential, 
Mr instance, reading at the national norm for grade'level, then cost be- 
comes the principal factor in choosing among programs which reach this 
cichipvemont level or above. 

At this point the cost-effectiveness model and Its components have 
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been described. However, It 1r one thing to develop an Ideal ^ theoretical 
model In the abstract and quite •-another co apply that model In a real sit-* 
uation. In the next section^ some preliminary attempts to apply one as-- 
ppct of the model through collecting primary cost data from faculty will 
be described. 

Gene ral Competency Assessment by Faculty 

There were two specific purposes In this portion of the Competency 
Assessment study. First, examination of how effectively competencies 
were presently being assessed within the regular courses In a library 
S(!hool and In a school of education needed to he determined. Secondly^ 
some baseline cost data needed to be collected both to test the primary 
ii:ost matrix of our model and also to begin the process of building an 
empirical data bank of cost information. This section of the study was 
guided by four research questions: 

(1) What is the distribution of competencies within 
courses? /' 

(2) What is the distribution of assessment strategies 
within courses? 

(3) What is the cost of course-related assessment 
activities? 

(i) How is the effectiveness of assessment strategies 
to be determined? 

Erich of these questions will be discussed In sequence. 

y— ) 

Distribu tion of Com potencles within Courses . A year and a half 
oarllpr all tlie faculLv of the School of Information Studies and. of the 
Area of Instruct Ion. il Tethnology, Schoo^l of Education wer.3 interviewed 
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to determine which competencies were taught in their clashes In order to 
advise students which particular set of courses to take. Advisement fol- 
lowed a thorough diagnosis of each student's present level of skills and 
abilities with a joint determination of where the student needed to be- 
come more proficient. After the determination of a desired set of com- 
pecencies to be acquired or added to, the courses were then examined to 
see which group of courses would be the most appropriate for the student. 

Dur/lng this part of the project, the researchers went back to the 
faculty to ask them not just which competencies they taught, but which 
ones were actually assessed and in what manner the assessment took place. 
Outlines, activities, and assignments were collected from faculty for 
each relevant course and then classified by types of , assessment proce- 
dures. The interviews were analyzed and a new competency-by-course 
m;,trix drawn (see Figure 8). Not surprisingly, it was found that the 
fdoulty teach more competencies than they actually assess. 

A basic assumption that guided the work throughout the prcject is 
that Che school media competency program must fit within a larger library 
school context composed of courses, many of which are general in nature, 
and directed to students who will work in a variety of library settings. 
Thv importance of this approach should be underscored. The alternative 
aHstimption would be that the school media program would be treated as a 
closed svstcm isolated and apart. Under this assumption, there would be 
a school within a school with a set of courses tailored to fit the 
sauH)! cnmpetency-based program, and which prospective school media 
sptH tn lists would take by themselves away from the rest of the school. 

The broader approach has a number of advantages. It allows school 
media specialist students to relate their work to the larger context. 
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Of equal or greater Importance, It enlarges faculty awareness of the 
special needs and contributions being made by the modern school library 
media center. This positive benefit more than offsets the requirement 
for regularly checking with all faculty—old and newfor changes In 
courses, new content, new methods, etc. Competency assessment then be- 
comes a dynamic process continually being adjusted to the changing Issues 
and the changing environment. 

Distribution of Assessment Strategies within Courses , In the re- 
interviews with faculty it was found that there were redundancies- 
many competencies taught and assessed from different points pf view in 
different courses. There were also some gaps where competencies were 
not assessed at all through formal courses. Figure 9 shows the ranking 
of frequency of assessment by competency number. Each year this analy- 
sis needs to be a part of the process. Those competencies not being ^ 
directly assessed through course work may be assessed through field 
work, independent projects, or on an individual or small group basis 
outside of courses. As a result of the analysis, suggestions will be 
made for course revision to include and/or to measure the attainment of 
other competencies where appropriate . 

Data did not always fall out as neatly as the researchers might 
have liked. In some cases the data revealed that the courses allowed 
students to demonstrate only a few of the competencies, while in other 
'Mses many competencies could be demonstrated. In all cases, however, 
the courses considered more than the competencies of interest to the 
project and the faculty were assessing more than just those competencies, 
This is important as It means the competency-based program sets a floor 
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rather than a celling. Many qualifying statements accompanied faculty 
\ responses. These statements explained the use of a given assessment 
technique. They emphasized the degree to which the technique assessed 
a given competency. And they revealed the Instructor's approach to 
assessment, which led to an exploration of the impact of differing 
faculty styles of assessment. 

Cost and Effectiveness of Course-Related Assessment Activities . It 
is necessary to collect specific cost and effectiveness data on a peri- 
odic basis and to compare results in order to make responsible resource 
allocations satisfying through sub-optimization the needs of the program, 
the school, and society. The researchers have carefully developed a 
model that specifies the universe of cost categories and effectiveness 
CJ Iteria. 

Three alternatives were considered for testing the cost model. It 
Ir? possible to look at costs of field work; secondly, an assessment 
situation could be established outside of classes through simulation, 
Laboratories and summative testing; or an attempt could be made to cost 
the assessment activity that was already going on in classes. The third 
alternative was the most appealing since y probably would have the 
broadest app) icability to other programs. 

Onlv primary cost information was collected and no attempts were 
made to grapple with social costs, psychological costs, lost opportunity 
costs, and the like. Nor would data be collected to determine degree of 
effectiveness at this time. A simplifying assumption was adopted, I.e., 
that assessment through course work was primarily a function of faculty 
t ime. Faculty t ime was assumed to be differentially allocated to assess- 
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ment activity based on: (1) whether or not the course had been taught 
previously, (2) the number of students in the coursei (3) the number of^ 
competencies being assessed in the course » and (4) the faculty style of 
assessment (including the number of assessment activities). 

Using these assumptions, the researchers asked faculty members to 
estimate the number of hours spent on assessment activity for a given 
course, the proportion of that assessment time dedicated to the speci- 
fied competencies relevant to the school media program, and the estimated 
accuracy of the estimates. * Figure 10 shows the data collected. 

It was djecided not to try to collect data retrospectively in the 
various function categories of test administration, test analysis, test 
recording-reporting, and test-related advisement, but rather to be satis- 
fled with a global measure, as it was felt that the accuracy level of 
the estimates was going to be somewhjSt low and it would further attenu- 
ate the accuracy to break estimates into more discrete categories- 

The faculty were also asked to estimate the costs of material used 
In the asfiessment activity and any other costs associated with assess- 
ment, for example, travel, graduate assistant time, equipment, etc. The 
time estimates were converted to dollars using a conservative factor 
(based oh average annual professorial salaries divided by an estimated 
fifty hour work week) of $15 per hour. Cost of materials and other costs 
wore also converted to dollar amounts. 

Figure 11 shows a sample of the results collected in this pilot test. 
The two courses shown were very different in nature. ITE 719 Is an ad- 
ministration course; 1ST 612 is a children's literature course. Although 
more competencies were assessed in the administration course, there were 
twice as many students in the literature course and nearly three times 
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the number of assessment activities undertaken. These factors account 
for most of the difference in total cost. The range of cost of the 
twerty-four cotirses for which data was collected varied widely although 
the mean and median measures were fairly close. 

Obviously, 9nly .limited conclusions can be reached from this small 
retrospective data gathering effort. Many more factors would need to 
be included and real-time measures applied. The exerclfse did, however, 
indicate the feasibility of the approach and the willingness of the 
total faculty to assist In the data collection effort. 

Elaboration of the Cost-Effectiveness Model 

Several specific areas are in need of elaboration; 

(1; The specific procedures for collecting and calculating cost 
data need greater specification. Especially needed are tech- 
niques for estimating average time and usage estimates, trans- 
lating traditional budgets to function and resource cate- 
gories, and deriving program specific costs. 

(2) The rel.itlonshi|) of the effectiveness model to test valida- 
tion procedure.s needs to be thoroughly explored. Validation 
iirocedures for non-pencll-and-paper tests need to be identl- 
fleii and/or developed. 

(3) The implied correlations between strategies and degree of 

ef fectlvenesK In assessing competence must be experimentally 
validated. 
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